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Concept map structure, testing, and scoring methods are discussed and a new scoring 
methodology is introduced using the breadth and depth of individual concept maps. The scoring 
method proposed here provides advantages of grading “on a curve” such as the ability to 
estimate and compare the complexity of different concept maps, the ability to measure the 
statistical distance between individual map scores, the ability to define map-score quantiles, a 
reduction in grading bias, and an overall improvement in the reliability and validity of concept 
map evaluation. Empirical support for underlying probability distributions is presented and 
mathematical aspects are discussed. Tables of concept map score distributions for one branch 
and one node are provided and may be used to very quickly assign concept map scores based on 
one branch and one node. 
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A concept map is a two-dimensional hierachical diagram that reflects how 
knowledge is organized. Concept maps are often used to evaluate science and math knowledge 
and are generally accepted as viable evaluation and research instruments. Concept maps may 
also be used as pre- assessment and post-assessment instruments, to assist in the clarification, 
consolidation, and reinforcement of knowledge, and in sequencing concepts in lessons and 
curricula. 



A concept map hierarchy has the most general concepts in the center (or at the top) 
and has branches that extend outward to the most specific concepts. Because of the hierarchical 
structure, concept maps reveal students’ understandings of relationships between concepts in 
various areas and provide an alternative to traditional testing instruments. Moreover, concept 
maps measure a dimension of knowledge that is not usually assessed by traditional tests. 

STRUCTURE 

Concept maps might contain only concepts and connecting links (or arrows). 




Concept maps might also contain propositions on the links between concepts. 





TESTING 



-A. type of function 



Polynomial 



A. type of* polynomial 




Concept map tests range from formats where students start with a blank piece of 
paper, to formats where lists of concept and relating propositions are provided, to where students 
start with a pre-arranged map structure with designated spaces for concepts and. In all cases, a 
key or base concept is provided. 



function 




Students develop a list of related concepts from memory, from class notes and/or text book, or 
students may be given a list of concepts and/or propositions. Usually, the concepts are then 
organized into groups ranging from the most general, inclusive concepts to the most specific, 
least inclusive concepts. 

SCORING 



Typically, scores assigned to concept maps are based on the number of times various 
map features correctly appear in the maps. For example, the number of concepts, the number of 
hierarchy levels, the number of cross-links, the number of propositions, and/or the number of 





examples can all be use to compute a score. To help with the scoring process, the scorer might 
refer to lists of concepts, cross-links, propositions, and examples derived from expert-prepared 
maps so that a score reflect the similarity between the graded map and an expert map. After 
frequencies are found, the numbers might be weighted to give more credit to some features over 
others. For example, features might be weighted by 1 for each concept, 1 for each proposition, 2 
for each example, 3 for the number of hierachy levels, and 3 for each cross-link. The weighted 
numbers are then added together to get a total performance score. 

Figure 1. 




The map in Figure 1 has eight concepts (not counting the base concept), zero 
propositions, zero examples, two hierarchy levels, and two crosslinks. Assuming all features are 
correct, the score for this map would be 1X8 -i- 1X0 -i- 2X0 -i- 3X2 -i- 3X2 = 20. One problem with 
this scoring methodology, however, is that highly dissimilar maps can end up with similar 
scores. For example, the map in Figure 2 has five concepts (not counting the base concept), zero 
propositions, zero examples, five hierarchy levels, and zero crosslinks. Assuming all features are 
correct, the score would be 1X5 -i- 1X0 -i- 2X0 -i- 3X5 -i- 3X0 = 20. 



Figure 2. 




A SCORING METHODOLOGY 



After observing many concept maps drawn by many students, there are two concept 
map features that stand out in their ability to reflect the level of understanding shown in the 
concept map construction. These two features are the breadth of the map and depth of the map. 
That is, students with a better understanding of a particular domain tend to draw maps with a 
greater number of branches and a greater average branch length than do students with a poorer 
understanding in the domain. Thus, a straightforeward way to grade concept maps is to base the 
score on the average number of branches eminating from nonterminal concept nodes, and the 
average branch length. For example. Figure 3 shows how the average number of branches 
eminating from nonterminal concept nodes would be calculated for a particular concept map. 

Figure 3. 




score=(6+ 1+4+ 1+1+ 1+1+ 1+2+4+4)711 = 2.36 branches per nonterminal concept 



The greatest advantage in using the average number of branches eminating from 



nonterminal concept nodes together with the average branch length is that relative performance 
information becomes readily available. Specifically, relative performance scoring methods 
assume there is an underlying probability distribution that describes the concept maps scores. 
Even though debates continue as to whether relative-performance grading methods are better 
than performance-based methods in general, the advantages of grading concept maps “on a 
curve” are many. The relative performance method described below gives the ability to estimate 
and compare the complexity of different concept maps, the ability to measure the statistical 
distance between individual map scores, the ability to define map-score quantiles, a reduction in 
grading bias, and an overall improvement in the reliability and validity of concept map 
evaluation. 



When concept maps with large numbers of both nonterminal and terminal concepts 
are scored based on the average number of branches eminating from nonterminal concepts and 
the average branch length, the conditions of the Central Limit Theorem are fairly well met. As 
the numbers increase, the distribution of the standardized scores approaches a normal probability 
distribution and all the advantages of “normed” grading method can be realized. However, if the 
numbers of nonterminal and terminal concepts are small, using a continuous normal distribution 
to model scores computed from discrete variables is not appropriate. Moreover, scoring large 
concept maps is generally not accurite and too time consuming. Instead, it is better to test 
students in very specific knowledge domains so that their maps are compact and manageable. 

In the nearly 40 years that concept maps have been used as knowledge-evaluating 
instruments, no advancements have significantly improved concept-map assessment. However, 
goodness-of-fit tests reveal that when concept maps are scored using the average number of 
branches eminating from nonterminal concepts and the average branch length, the scores are 
accurately described by a certain type of modified joint Poisson distribution. The theoretical 




argument for why a Poisson model describes the number of branches eminating from 
nonterminal concept nodes is as follows: 



WHY A POISSON DISTRIBUTION 

For the number of branches eminating from nonterminal concepts, consider all concepts 
that could be correctly linked to a specific node. Partition these concepts into classes of equal 
difficulty. That is, create sets of concepts of the same cognitive level. Call the sets difficulty 
classes. If n concepts are in a difficulty class, the probability that x out of n are linked to a 
specific node is given by a binomial probability model provided the following two conditions 
hold: 

(i) the chance of selecting any new concept is constant for all concepts (uniformity), and 

(ii) branching to a new concept is independent of any concept already in the map 
(independence). 

In general, condition (ii) above does not hold. However, if a linked concept within a 
difficulty class is counted only when no other linked concepts are hints for (correlated with) any 
other linked concept, then the independence assumption is reasonable. 

If the uniformity and independence assumptions hold, and if the number of concepts 
within each difficulty class is large, then, the Poisson distribution approximates the probability 
that X concepts are selected. Further, if the difficulty classes are independent, then the TOTAL 
number of concepts selected over ALL difficulty classes also has a Poisson distribution. 

An important property of the Poisson distribution is that the mean number of concepts is 
the sum of the mean number of concepts for each of the difficulty classes. More specifically, let 




Y1 + Y2 be the sum of the numbers of concepts from the 1st and 2nd difficulty levels. It follows 
that if Y1 ~ P(ml) (Piosson with mean=ml) and if Y2 ~ P(m2), then Y1 + Y2 ~ P(ml + m2). 

A similar argument could be made for branch lengths having a Poisson distribution. 
However, from an examination of concept map data it was found that the variance of branch 
length was less than the variance of the number of branches eminating from non-terminal nodes. 
Thus, a standard Poisson distribution (where the mean equals the variance) did not accurately 
describe the data.. The Conway-Maxwell-Poisson (CMP) distribution gave a fit that was better 
than the Poisson distribution because this distribution has a second parameter that affects the 
variance. But as with the Poisson distribution, the variance of the CMP distribution was still too 
large. A modification to a truncated CMP distribution that reduced the variance (see below) 
worked well in describing branch length and, moreover, only one parameter was needed. 

TRUNCATED DISTRIBUTIONS 

There is one more consideration in using Poisson and modified CMP models to 
describe concept map scores. A truncated Poisson model is used when the number of zero 
observations is unknown or unobserved, or if the population size is unknown. In the case of 
concept maps, the number of concepts that are NOT linked to a terminal node is unknown and 
unobserved, and in many cases, the number of concepts that might be linked to a given concept 
is unknown. 



Some other examples of when truncated Poisson models might be used are as 
follows. Consider the number of people with an infectious disease (e.g. measles) per household - 
- the number of people that do not get sick varies with household size (population size). Consider 
the number of accidents in a factory — the number of employees varies per factory, so the 
number who do not have accidents is unknown. Consider the number of butterflies caught in a 
net in one pass through a field — the number of butterflies not caught is unknown. Consider the 




number of words that are repeated once, twice, three times, etc. — the number of words repeating 
zero times is unknown. 



EMPIRICAL SUPPORT FOR TRUNCATED POISSON AND MODIFIED CMP MODELS 

The following five tables present the results of goodness-of-fit tests that show how closely 
truncated Poisson distributions fit the empirical distributions of the number of branches 
eminating from nonterminal concepts. 

The data in Table 1 show the number of links (branches) eminating from nonterminal concepts 
and the observed frequencies (for example, there were 29 nodes with 2 eminating branches) for 
students who completed concept maps in Class I. A chi-square test shows the empirical and 
truncated Poisson distributions are very close. One would have to accept a 70% chance of being 
wrong to conclude that the empirical and Poisson distributions were different. 

Table 1. 









Data 1 








No. links 


Obs Freq. 


Mean 


Lambda 


Trunc Poisson 


Exp Freq 


(o-e’"^/ e 


1 


51 






0.471116 


47 


0.32092 


2 


29 






0.319098 


32 


0.26535 


3 


12 






0.144089 


14 


0.40272 


4 


5 






0.048797 


5 


0.00296 


5 


2 






0.013220 


1 


0.34762 


6 


1 






0.002984 


0 


1.64867 
















Total 


100 


1.81 


1.3546 


Chi-sq= 


2.9882 


Alpha = 












(k-1) 
= 5 d.f. 


0.70 




60 




The data in Table 2 show the number of branches eminating from nonterminal concepts and the 
observed frequency of each number (for example, there were 2 nodes having 6(not shown) 
eminating branches) for students who completed concept maps in Class II. A chi-square test 
shows the empirical and truncated Poisson distributions are very close. One would have to 
accept a 88% chance of being wrong to conclude that the empirical and Poisson distributions 
were different. 



Table 2. 



Data II 



ObsFreq 


Mean 


Lambda 


Trunc 

Poisson 


Exp Freq 


(o-e*"^/e 


47 






0.357616 


51 


0.28159 


52 






0.320953 


46 


0.90565 


26 






0.192032 


27 


0.05902 


11 






0.086172 


12 


0.12495 


4 






0.030935 


4 


0.03512 


2 






0.009254 


1 


0.35793 














142 


2.1478 


1.7949 


Chi-sq = 


1.7642 


Alpha 










(n-1) = 5 d.f. 


=0.88 



60 



45 



Poisson 
frequency 
= np 



30 - 



observed frequency 



IT 



2 3 4 

number of branches 



The data in Table 3 show the number of branches eminating from nonterminal concepts and the 
observed frequency of each number (for example, there were 5 nodes having 4 eminating 
branches) for students who completed concept maps in Class III. A chi-square test shows the 
empirical and truncated Poisson distributions are very close. One would have to accept a 91% 
chance of being wrong to conclude that the empirical and Poisson distributions were different. 



Table 3. 









Data 1 1 1 








No. 

links 


Obs Freq 


Mean 


Lambda 


Trunc 

Poisson 


Exp Freq 


(o-e’"^/ e 


1 


24 






0.4223 


23 


0.062380 


2 


16 






0.3235 


17 


0.123623 


3 


8 






0.1651 


9 


0.095032 


4 


5 






0.0632 


3 


0.733935 


5 


1 






0.0193 


1 


0.002089 






























Total 


54 


1.944 


1.5319 


Chi-sq = 


1.0170 


Alpha= 












(n-1) =4df 


0.91 




The data in Table 4 show the number of branches eminating from nonterminal concepts (x) and 
the observed frequency of each number (for example, there were 16 nodes having 2 eminating 
branches) for students who completed concept maps in Class IV. A chi-square test shows the 



empirical and truncated Poisson distributions are very close. One would have to accept a 30% 
chance of being wrong to conclude that the empirical and Poisson distributions were different. 



Table 4. 





X 


Poisson 


exp freq 


obs freq 


(o-e’"^/e 


x-bar= 












1.70149 


1 


0.514071 


34 


39 


0.60298 




2 


0.311063 


21 


16 


1.124577 




3 


0.125482 


8 


6 


0.689302 




4 


0.037965 


3 


5 


2.372122 


lambda= 


5 


0.009189 


1 


1 


0.23994 


1.2102 
























N=67 






chi-sq= 


5.02892 


alpha=0.30 



POISSON & OBSERVED FREQUENCIES vs NUMBER OF BRANCHES 



40 



30 



- Poisson 
freq 
= np 



observed frequency 



S' 20 



10 



12 3 4 

number of branches 



The data in Table 5 show the number of branches eminating from nonterminal concepts (x) and 
the observed frequency of each number (for example, there were 58 nodes having 1 eminating 
branch) for students who completed concept maps in Class V. A chi-square test shows the 
empirical and truncated Poisson distributions are very close. One would have to accept a 17% 
chance of being wrong to conclude that the empirical and Poisson distributions were different. 

Table 5. 





X 


Poisson 


expfreq 


obs 

freq 


(o-e’"^/e 


x-bar= 












1.73 


1 




50 


58 


1.196113 




2 


0.313601 


31 


23 


2.22867 




3 


0.130482 


13 


11 


0.3215 




4 




4 


5 


0.211608 




5 


0.010165 


1 


2 


0.951581 




6 


0.002115 


0 


1 


2.940282 


Lambda = 










1.248226 












n=99 




chi-sq = 


7.849754 


k-1 = 
5d.f. 


alpha= 

0.17 

















POISSON & OBSERVED FREQUENCIES vs NUMBER OF BRANCHES 



60 



45 - 



30 - 



15 - 



Poisson 

frequency 

= np 



observed frequency 






2 3 4 

number of branches 



BRANCH LENGTH 

The following two tables present the results of goodness-of-fit tests that show how closely 
truncated and modified CMP distributions fit branch length data collected from two classes. 

The data in Table 6 show branch lengths and the observed frequency of each length (for 
example, there were 1 17 branches of length 2) for students who completed concept maps in 
Class VI. A chi-square test shows the empirical and truncated Poisson distributions are close. 
One would have to accept a 12.4% chance of being wrong to conclude that the empirical and 
modified CMP distributions were different. 



Table 6. 



Branch 


Ohs 








Exp 




Length 


Freq 


Mean 


Lambda 


Dist 


Freq 


(o-e)^2/e 


1 


58 






0.3 


66 


0.88 


2 


117 






0.47 


103 


1.93 


3 


32 






0.19 


41 


2.02 



4 10 

5 0 

Total= 217 1.97 



1.57 



0.03 7 0.92 

0 0 0 

Chi- 

sq= 5.76 Alpha = 
3 

degrees 0.124 
of 

freedom 




1 2 3 4 5 

branch length 



The data in Table 7 show branch lengths and the observed frequency of each length (for 
example, there were 10 branches of length 4) for students who completed concept maps in Class 
VII. A chi-square test shows the empirical and truncated Poisson distributions are close. One 
would have to accept a 11.8% chance of being wrong to conclude that the empirical and 
modified CMP distributions were different. 



Table 7. 



Branch 

Length 


Ohs 

Freq 


Mean Lambda 


Dist 


Exp 

Freq 


(o-e)^2/e 


1 


58 




0.28 


72 


2.58 


2 


121 




0.48 


122 


0.01 



3 


61 






0.2 


49 


2.83 


4 


10 






0.03 


9 


0.24 


5 


2 






0 


1 


1.69 


Total= 


256 


2.09 


1.72 


Chi-sq= 


7.35 

4 

degrees 

of 

freedom 


Alpha = 
0.118 




branch length 



MATHEMATICAL BACKGROUND 



The complete Poisson distribution may be expressed as 



-Xr. X 

f(x, A.) = - — ; — ,x = 0,1,2,... 



X! 



Where is the probability of observing exactly x occurrences of the event under 

A 



consideration and 



is the mean and variance of the number of occurrences. 



The probability of zero occurrences is 



f(0, X) = Q 



so the probability of one or more occurrences is 




If x=0 is not being considered, the remaining probabilities for x=l, 2, ... are normali z ed by 
multiplying by 



1 



1 






- e 



Then the Poisson distribution, truncated below 1 is 



f(x, X) 



-X. X 

e X 
x! 



1 - e 






, X 




Or 



f(x, X) 




X 




Then 



cx> 



Z 

X = 1 





1 ) 



1 




and 



f(x, X) _ X 
f(x - l,X) X 

and, just as with the complete Poisson distribution 

f(x, X) = -f(x - 1, A,) 

X 



The shape of the truncated Poisson distribution is shown in the figure below for various values of 
lambda. 



TRUNCATED POISSON DISTRIBUTIONS: lambda = 0.6, 2.9, 4.9 




Parameter Estimation 



V 

If is the mean of a sample from a Poisson distribution with the zero class missing, then the 
maximum likelihood estimate of the population mean is 









then the maximum likelihood estimate of the population mean is 

-X J 

(xe 

i! ^ 

j = 1 



00 J - 1 
j“ 






Or 







The CMP distribution is given by 



f\y^ V) = 






(y!) 



00 

V 



A 






y = 0,1.2.... 



The truncated CMP distribution is given by 






1 



= 1 , 2 ,... 






(>^!) 



V 00 



X 






The joint distribution has the form 



f(x, y, X^, X2, V, c) = 






A 



V 



x!(j!) 



V 



X normalizing 



term 



Where X has a truncated Poission distribution, Y has a truncated CMP distribution, and c is the 
covaiance of X and Y. Note that because the joint distribution must be positive, c must be less 
than the minimum of lambdai and lambda 2 . It is clear that the number of branches and branch 
length are positively correlated but the standard error of the covariance c may be small for all 
math and science concept maps. The scoring process would be simplified if a universal constant 
could be used for c. This is being investigated by the author but for now, the population 
covariance should be computed in the traditional way. 

The above distribution may be used to construct tables of cummulative score 
distributions. Concept map scores may be assigned based on the cummulative values of the 
above joint truncated distribution for various population mean branch length and population 
mean number of branch values are given in the tables below. The tables below give cummulative 
percentages for the joint Poisson / Conway-Maxwell-Poisson distribution. Tables for sums of 
branches from nodes, and sums of branch lengths (that may be used to assign more valid concept 
map test scores) are being investigated. The tables below may be used to quickly assign scores 
for individual concept maps based on one branch and one node selcted from each map. 



CONCEPT MAP CUMMULATIVE PROBABILITY TABLES (for one node and one branch) 



P(X<=x andY<=y) 

Mean number of branches (X-bar) = 1.6 

Mean branch length (Y-bar) = 1 .6 

Covariance 

(X,Y) = 0.2 

length of one branch y 







1 


2 


3 


4 


5 


6 




1 


47 


60 


62 


62 


62 


62 


number 


2 


67 


86 


89 


90 


90 


90 


of 


3 


73 


94 


97 


98 


98 


98 


branches (one node) 


4 


74 


96 


99 


99 


99 


99 


X 


5 


75 


96 


99 


100 


100 


100 




6 


75 


96 


99 


100 


100 


100 



Ave number of branches (X-bar) = 1 .6 




Mean branch length (Y-bar) = 


1.6 












Covariance 














(X,Y) = 


0.4 














length of one branch y 








1 2 


3 


4 


5 


6 




1 


56 68 


70 


70 


70 


70 


number 


2 


75 92 


93 


94 


94 


94 


of 


3 


79 97 


99 


99 


99 


99 


branches (one node) 


4 


80 98 


100 


100 


100 


100 


X 


5 


80 98 


100 


100 


100 


100 




6 


80 98 


100 


100 


100 


100 


Ave number of branches (X-bar) = 


1.6 








Mean branch length (Y-bar) = 


1.6 












Covariance 














(X,Y) = 


0.6 














length of one branch y 








1 2 


3 


4 


5 


6 




1 


66 77 


78 


78 


78 


78 


number 


2 


82 95 


96 


96 


96 


96 


of 


3 


85 98 


99 


99 


99 


99 


branches (one node) 


4 


85 98 


99 


100 


100 


100 


X 


5 


85 98 


99 


100 


100 


100 




6 


85 98 


99 


100 


100 


100 



Ave number of branches (X-bar) = 
Mean branch length (Y-bar) = 
Covariance 
(X,Y) = 

1 2 


1.6 

1.6 

0.8 

length of one branch y 

3 4 


5 


6 


1 


79 86 


87 


87 


87 


87 


number 2 


90 98 


99 


99 


99 


99 


of 3 


91 99 


100 


100 


100 


100 


branches (one node) 4 


91 99 


100 


100 


100 


100 


X 5 


91 99 


100 


100 


100 


100 


6 


91 99 


100 


100 


100 


100 



Ave number of branches (X-bar) = 


1.6 




Mean branch length (Y-bar) = 


1.6 




Covariance (X,Y) 


1 

length of one branch y 




1 2 


3 4 5 


6 


1 94 96 


96 96 96 


96 




number 


2 


97 


100 


100 


100 


100 


100 


of 


3 


97 


100 


100 


100 


100 


100 


branches (one 
node) 


4 


97 


100 


100 


100 


100 


100 


X 


5 


97 


100 


100 


100 


100 


100 




6 


97 


100 


100 


100 


100 


100 



Ave number of branches (X-bar) = 
Mean branch length (Y-bar) = 
Covariance 
(X,Y) = 

1 2 


1.8 

1.6 

0.2 

length of one branch y 

3 4 


5 


6 


1 


40 51 


53 


53 


53 


53 


number 2 


63 81 


84 


84 


84 


84 


of 3 


71 92 


95 


95 


95 


95 


branches (one node) 4 


74 95 


98 


99 


99 


99 


X 5 


74 96 


99 


99 


99 


99 


6 


75 96 


99 


100 1 


00 


100 



Ave number of branches (X-bar) = 
Mean branch length (Y-bar) = 
Covariance 
(X,Y) = 

1 2 


1.8 

1.6 

0.4 

length of one branch y 

3 4 


5 


6 


1 


48 59 


60 


60 


60 


60 


number 2 


71 86 


88 


88 


88 


88 


of 3 


78 95 


97 


97 


97 


97 


branches (one node) 4 


79 97 


99 


99 


99 


99 


X 5 


80 97 


99 


100 


100 


100 


6 


80 97 


99 


100 


100 


100 



Ave number of branches (X-bar) = 
Mean branch length (Y-bar) = 
Covariance 
(X,Y) = 

1 2 


1.8 

1.6 

0.6 

length of one branch y 

3 4 


5 


6 


1 


58 67 


67 


68 


68 


68 


number 2 


79 91 


93 


93 


93 


93 


of 3 


84 98 


99 


99 


99 


99 


branches (one node) 4 


85 99 


100 


100 


100 


100 


X 5 


86 99 


100 


100 


100 


100 


6 


86 99 


100 


100 


100 


100 




Mean number of branches (X-bar) = 1 .8 

Mean branch length (Y-bar) = 1 .6 

Covariance (X,Y) 

0.8 



length of one branch y 







1 


2 


3 


4 


5 


6 




1 


69 


75 


76 


76 


76 


76 


number 


2 


88 


96 


96 


96 


96 


96 


of 


3 


91 


99 


100 


100 


100 


100 


branches (one 
















node) 


4 


92 


100 


100 


100 


100 


100 


X 


5 


92 


100 


100 


100 


100 


100 




6 


92 


100 


100 


100 


100 


100 


Ave number of branches (X- 


bar) = 


1.8 








Mean 


branch length (Y 


-bar) = 


1.6 
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